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Preface 


This  report  is  1  of  6  interdependent  research  outlines  in  the  Adaptive  Training 
research  program.  Portions  of  this  text,  which  originated  in  ARL-SR-0325, 1  appear 
in  all  6  reports  to  ensure  that  readers  get  the  same  cross-cutting  infonnation. 


*  Sottilare  R,  Sinatra  A,  Boyce  M,  Graesser  A.  Domain  modeling  for  adaptive  training  and  education  in 
support  of  the  US  army  learning  model-research  outline.  Aberdeen  Proving  Ground  (MD):  Army  Research 
Laboratory  (US);  June  2015.  Report  No.:  ARL-SR-0325. 
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1.  Introduction 


Training  and  education  tools  and  methods  must  be  of  sufficient  intelligence  to 
understand  the  needs  of  individual  learners  and  units  of  learners  to  mitigate 
negative  learner  states  and  to  guide  and  tailor  instruction  in  real  time  to  optimize 
learning.  These  tools  and  methods  must  also  be  affordable,  effective,  and  easy  to 
access  and  use.  These  requirements  are  enablers  of  the  US  Army  Learning  Model 
(ALM),  which  includes  an  emphasis  on  self-regulated  learning  (SRL),  where 
Soldiers  are  expected  to  manage  their  own  learning  and  career  development  through 
the  growth  of  metacognitive  (e.g.,  reflection),  self-assessment,  and  motivational 
skills  (Butler  and  Winne  1995).  While  SRL  skills  are  difficult  to  train  and  develop, 
support  may  be  provided  to  the  learner  through  “adaptive  training  technologies” 
(tools  and  methods)  that  may  be  focused  to  guide  learning  and  reinforce  SRL 
principles. 

To  support  ALM,  the  US  Army  Research  Laboratory  (ARL)  has  developed  a 
program  of  research  called  “adaptive  training”  that  includes  6  interdependent 
research  areas  or  vectors:  individual  learner  and  unit  modeling,  instructional 
management  principles,  domain  modeling,  authoring  tools  and  methods,  evaluation 
tools  and  methods,  and  architectural  and  ontological  support  for  adaptive  training. 
The  reports  documenting  these  vectors  expand  the  scope  of  the  adaptive  tutoring 
research  described  in  ARL-SR-0284  (Sottilare  2013)  to  support  ALM  requirements 
in  the  mid-  and  long-term  evolution  of  training  and  educational  technology:  the 
Synthetic  Training  Environment  and  the  Future  Holistic  Training  Environment  for 
Live  and  Synthetic. 

This  report  (1  of  6  interdependent  research  outlines)  focuses  on  instructional 
management  research  for  adaptive  training  and  education.  An  overarching  goal  of 
instructional  management  is  to  leverage  prior  research  in  the  learning  sciences  and 
artificial  intelligence  (AI)  communities  (both  theoretical  and  empirical)  to  establish 
a  set  of  best  practices  on  how  to  author  and  execute  techniques,  strategies,  and 
tactics  across  any  domain  of  instruction.  This  is  a  very  large  research  space  (Wang- 
Costello  et  al.  2013),  so  it  is  important  to  identify  the  primary  themes  for  applied 
instructional  management  practices. 

Today  the  majority  of  intelligent  tutoring  systems  (ITSs),  a  form  of  adaptive 
training  tool  to  support  one-to-one  computer-based  instruction,  supports  well- 
defined  domains  in  mathematics,  physics,  and  software  programming.  Since 
Soldiers  operate  in  more-complex,  more-dynamic,  and  more-ill-defined  domains, 
it  is  necessary  to  expand  the  scope  of  adaptive  training  tools  and  methods  to  support 
training  and  education  in  these  militarily  relevant  task  environments.  Instructional 
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management  in  adaptive  training  environments  represents  pedagogical  logic  and 
policies  that  dictate  what  is  experienced  and  what  assistance  is  provided.  The  goal 
is  to  align  instructional  management  techniques  with  empirical  research  that 
informs  what  strategies  have  the  greatest  influence  on  performance,  retention,  and 
transfer.  Effective  instructional  management  research  will  dictate  how  best  to 
instruct  a  particular  task  or  set  of  concepts  for  a  given  type  of  learner  that  promotes 
the  development  of  knowledge  and  skill  across  any  domain  of  instruction. 

2.  Research  Goals  and  Objectives 

The  goal  of  the  research  described  in  this  report  is  to  model  militarily  relevant 
training  domains  to  support  individually  tailored  and  intelligently  guided  training 
experiences  as  prescribed  by  the  ALM  (US  Anny  Training  and  Doctrine  Command 
2011).  The  research  provides  guidelines,  best  practices,  tools,  models,  and  methods 
in  support  of  this  research  goal.  More  specifically,  we  intend  to  accomplish  the 
following: 

•  Discover  and  deliver  personalized  and  adaptive  training  experiences  that  are 
informed  by  experimentation  or  prior  research  on  tutoring  methods  and 
techniques  that  are  moderated  by  individual  differences  of  a  learner,  or  team 
of  learners,  to  promote  efficient  knowledge  and  skill  acquisition  across 
technology-driven  training  applications. 

•  Research,  design,  and  develop  prototype  authoring  tools  grounded  in 
learning  and  instructional  theory  and  informed  by  empirical  research  to 
assist  training  managers,  developers,  and  subject  matter  experts  (SMEs)  in 
building  pedagogically  sound  training  experiences  without  the  requirement 
of  programming. 

•  Discover  and  develop  modeling  functions  that  account  for  uncertainty 
across  policies  informing  pedagogical  decisions  (e.g.,  content  delivery, 
course  navigation,  and  guidance)  with  an  ability  for  these  functions  to  refine 
and  optimize  themselves  through  reinforcement  learning  mechanisms  over 
time  as  new  interaction  and  performance  data  becomes  available  (e.g., 
Markov  Decision  Processes). 

In  this  research  outline,  we  present  the  goals  associated  with  the  instructional 
management  research  vector  and  review  specific  subvectors  across  the  program  that 
influence  the  approach  and  direction  of  our  dedicated  efforts.  This  outline 
emphasizes  desired  functions  and  capabilities  for  ARL’s  Generalized  Intelligent 
Framework  for  Tutoring  (GIFT),  an  open-source  architecture  intended  to  support 
adaptive  instruction. 
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3.  Background 


While  human  tutoring  and  mentoring  are  common  teaching  tools,  current  US  Army 
standards  for  training  and  education  are  group  instruction  and  classroom  training, 
also  known  as  one-to-many  instruction.  Group  instruction  and  classroom  training 
have  been  generally  focused  on  acquiring  and  applying  knowledge  in  proxies  for 
live  training  environments  (e.g.,  desktop  simulations,  virtual  simulations, 
constructive  simulations,  and  serious  games). 

Classroom  training,  especially  for  complex  topics,  is  often  taught  as  a  series  of  lists 
that  the  instructor  goes  through  in  a  linear  fashion  (Schneider  et  al.  2013).  This 
approach  puts  a  heavy  burden  on  the  learner  to  build  mental  models  and  make 
conceptual  connections.  Using  this  instructional  methodology  may  lead  to  varying 
degrees  of  success  due  to  individual  differences  in  skills,  traits,  and/or  preferences. 
In  this  fashion,  lower-performing  students  who  struggle  are  provided  more  hands- 
on  guidance  while  higher-performing  students  who  are  under-challenged  (e.g., 
bored)  are  allowed  to  progress  through  a  course  at  an  accelerated  rate. 

Small  group  instruction  in  live  environments  has  also  been  used  to  assess 
application  of  knowledge  and  the  development  of  skills.  A  standard  feedback 
mechanism  for  US  Army  training  is  the  after-action  review  (AAR),  where 
significant  decision  points  and  actions  are  captured  for  small  group  discussions 
conducted  after  the  completion  of  a  training  event  to  help  capture  teachable 
moments  and  to  aid  Soldiers  in  reflecting  on  their  recent  training  experiences. 

Both  classroom  training  and  small  group  instruction  are  manpower-intensive, 
requiring  teachers,  mentors,  and  support  staff  to  guide  the  Soldier’s  experience. 
Today  ITSs  guide  learner  training  and  education  primarily  in  physically  static 
(e.g.,  desktop  simulations),  well-defined  domains.  Research  is  needed  to  expand 
ITS  instructional  management  practices  to  support  militarily  relevant  domains  that 
train  across  knowledge,  skills,  and  abilities  required  to  be  successful  Soldiers.  The 
focus  of  this  research  is  to  identify  pedagogical  principles  that  can  be  applied  to 
manage  knowledge  and  skill  acquisition  by  addressing  individual  differences, 
managing  affective  states  experienced  during  learning,  supporting  metacognitive 
development,  enforcing  reflection  through  personalized  AARs,  and  promoting 
retention  and  transfer  to  operational  environments. 

3.1  Self-Regulated  Learning  and  the  US  Army  Learning  Model 

In  20 1 1  the  US  Army  placed  significant  emphasis  on  the  development  of  SRL  skills 
with  the  expectation  that  new  methods  of  instruction  (e.g.,  ITSs)  would  augment 
institutional  training  (i.e.,  classroom  and  small  group  instruction). 
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One-to-one  human  tutoring  has  been  shown  to  be  significantly  more  effective  than 
one-to-many  instructional  methods  (e.g.,  traditional  classroom  instruction)  (Bloom 
1984;  VanLehn  2011).  However,  it  is  neither  practical  nor  affordable  to  have  one 
expert  human  tutor  to  mentor  each  Soldier  in  the  US  Anny  for  every  required 
operational  task.  This  alone  signals  the  need  for  capabilities  to  support  one-to-one, 
tailored  training  and  educational  experiences. 

Additionally,  under  the  ALM,  Soldiers  are  largely  responsible  for  managing  their 
own  learning,  but  SRL  skills  are  difficult  to  train  and  develop  (Azevedo  et  al.  2009; 
Butler  and  Winne  1995;  Graesser  and  McNamara  2010).  We  anticipate  adaptive 
training  tools  and  methods  will  fill  this  gap  and  provide  personalized  guidance  to 
acquire,  apply,  retain,  and  transfer  knowledge  and  skills  to  the  operational 
environment.  This  signals  the  need  for  a  computer-regulated  learning  strategy  to 
augment  missing  SRL  skills;  however,  adaptive  training  technologies  must  first 
become  affordable,  sufficiently  adaptive,  and  easy  to  use  for  this  strategy  to  be 
realized. 

3.2  Motivation  for  Research 

A  promising  alternative  to  one-to-one  human  tutoring  is  one-to-one  adaptive 
training  tools  that  include  ITSs.  Meta-analyses  and  reviews  support  the  claim  that 
ITS  technologies  routinely  improve  learning  over  classroom  teaching,  reading 
texts,  and/or  other  traditional  learning  methods.  These  meta-analyses  normally 
report  effect  sizes  (sigma  [o]),  which  refer  to  the  difference  between  the  ITS 
condition  and  a  control  condition  in  standard  deviation  units.  The  reported  meta¬ 
analyses  show  positive  effect  sizes  that  vary  from  a  =  0.05  (Dynarsky  et  al.  2007) 
to  a  =  1.08  (Dodds  and  Fletcher  2004)  but  most  hover  between  a  =  0.40  and 
a  =  0.80  (Fletcher  2003;  Graesser  et  al.  2012;  Ma  et  al.  in  press;  Steenbergen-Hu 
and  Cooper  2013,  2014;  VanLehn  2011).  Our  current  best  meta-meta  estimate  from 
all  of  these  meta-analyses  is  a  =  0.60.  This  performance  is  comparable  to  human 
tutoring,  which  varies  from  between  a  =  0.20  and  a  =  1.00  (Cohen  et  al.  1982; 
Graesser  et  al.  2011)  depending  on  the  expertise  of  the  tutor.  Human  tutors  have 
not  varied  greatly  from  ITSs  in  direct  comparisons  between  ITS  and  trained  human 
tutors  (Olney  et  al.  2012;  VanLehn  2011;  VanLehn  et  al.  2007). 

Graesser  et  al.  (2015  in  press)  are  convinced  that  some  subject  matters  will  show 
higher  effect  sizes  than  others  when  comparing  any  intervention  (e.g.,  computer 
trainers,  human  tutors,  group  learning)  with  a  control.  It  is  difficult  to  obtain  high 
effect  sizes  for  literacy  and  numeracy  because  these  skills  are  ubiquitous  in 
everyday  life  and  habits  are  automatized.  For  example,  Ritter  et  al.  (2007)  reported 
that  Cognitive  Tutor  software  for  mathematics  has  shown  an  effect  size  of  a  =  0.30 
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to  0.40  in  environments  with  minimal  control  over  instructors.  Human  interventions 
to  improve  basic  reading  skills  typically  report  an  effect  size  of  a  =  0.20.  In  contrast, 
when  the  student  starts  essentially  from  ground  zero,  such  as  many  subject  matters 
in  science  and  technology,  effect  sizes  are  expected  to  be  more  robust.  ITSs  show 
effect  sizes  of  a  =  0.60  to  2.00  in  the  subject  matters  of  physics  (VanLehn  2011; 
VanLehn  et  al.  2005),  computer  literacy  (Graesser  et  al.  2012;  Graesser  et  al.  2004), 
biology  (Olney  et  al.  2012),  and  scientific  reasoning  (Halpem  et  al.  2012;  Millis  et 
al.  2011).  As  a  notable  example,  Digital  Tutor  software  (Fletcher  and  Morrison 
2012)  improves  information  technology  by  an  effect  size  as  high  as  a  =  3.70  for 
knowledge  and  a  =  1.10  for  skills.  The  effect  sizes  attributed  to  improved 
instruction  and  improved  domain  knowledge  have  not  been  separated  in  this 
analysis.  Such  large  effect  sizes  would  never  be  expected  in  basic  literacy  and 
numeracy. 

Overall,  these  are  promising  results  and  equate  to  an  increase  of  about  a  letter  grade 
improvement  over  traditional  classroom  instruction.  While  ITSs  are  a  promising 
technology  to  support  adaptive  training  for  individuals  in  well-defined  domains  like 
mathematics,  physics,  and  computer  programming,  the  US  Anny  requires  the 
ability  to  develop  and  exercise  Soldier  skills  in  more  ill-defined  domains  (e.g., 
leadership)  and  at  the  unit  level  (e.g.,  collaborative  learning  and  team  training). 
Developing  and  maintaining  the  ability  to  make  effective  decisions  under  stress  and 
in  complex  environments  is  also  desirable. 

Adaptive  systems  by  their  nature  require  additional  content  and  complexity  to 
support  tailored  learning  for  each  user  and  as  a  consequence  have  a  very  high 
development  cost,  a  major  barrier  to  adoption  by  the  US  Anny.  Adaptive  systems 
are  also  insufficiently  adaptive  to  support  tailored,  self-regulated  training  and 
educational  experiences  across  a  broad  spectrum  of  military  tasks  as  required  by 
the  ALM.  Today,  few  ITS  authoring  tools  are  generalized  across  all  of  the  domains 
requiring  training,  and  no  evaluation  criteria  or  standards  have  been  developed  to 
promote  reuse  and  interoperability  among  ITSs  (Sottilare  et  al.  2012b).  In  other 
words,  current  adaptive  systems  are  not  yet  intelligent  enough  to  support  the 
tailored  instruction  required  by  the  US  Army  in  the  breadth  of  domains  being 
trained,  but  there  is  a  stable  foundation  of  50  years  of  science  on  which  to  grow  an 
adaptive  training  and  education  capability  for  the  US  Anny. 

3.3  Adaptive  Training  and  Education  Definitions 

In  support  of  the  ALM  and  affordable  adaptive  training  and  educational  capabilities 
for  the  US  Army,  ARL  is  investigating  and  developing  adaptive  tools  and  methods. 
A  desired  end-state  is  the  automation  of  authoring  (creation)  processes,  instruction, 
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and  evaluation  of  computer-regulated  training  and  education  capabilities  to  help 
build  SRL  skills  and  support  mixed-initiative  interaction.  A  major  goal  within  this 
research  program  is  to  reduce  the  time/cost  and  knowledge/skill  required  to  author, 
deliver,  and  evaluate  adaptive  technologies  to  make  them  usable  by  a  larger 
segment  of  the  Army  training  and  educational  community. 

Adaptive  training  and  education  research  includes  elements  of  adaptive  tutoring, 
distributed  learning,  virtual  humans,  and  training  effectiveness  evaluation.  For 
additional  detail  on  research  specific  to  ITSs,  refer  to  ARL-SR-0284  (Sottilare 
2013).  The  following  definitions  are  provided  for  this  section  to  distinguish 
between  adaptive  training  and  education  elements  and  also  to  highlight  their 
relationships: 

Adaptive  Tutoring:  Also  known  as  intelligent  tutoring,  these  tailored  instructional 
methods  provide  one-to-one  and  one-to-many  computer-guided  experiences 
focused  on  optimizing  learning,  comprehension,  performance,  retention,  reasoning, 
and  transfer  of  knowledge  and  acquired  skills  to  the  operational  environment. 

Adaptive  Tutoring  Systems:  Also  known  as  ITSs,  these  mechanisms  or 
technologies  (tools  and  methods)  provide  tailored  training  and  educational 
experiences.  Adaptive  tutoring  systems  respond  to  changing  states  in  the  learner 
and  changing  conditions  in  the  training  environment  to  optimize  learning  and 
anticipate  and  recognize  teachable  moments. 

Virtual  Humans:  Artificially  intelligent  visual  representations  of  people  that 
simulate  or  emulate  cognitive,  affective,  physical,  and  social  processes. 

Distributed  Learning:  Concurrent  distribution  of  training  and  educational  content 
to  multiple  users  at  the  point  of  need  in  which  content  is  intelligently  selected  to 
support  learning,  increased  performance,  and  long-tenn  competency  in  selected 
domains. 

Training/Learning  Effectiveness:  Evaluation  of  the  impact  of  training  and 
educational  tools  and  methods  on  usability,  learning,  comprehension,  performance, 
retention,  reasoning,  and  transfer  of  knowledge  and  acquired  skills  to  the 
operational  environment. 

Adaptive  Training  and  Education  Systems:  A  convergence  of  ITSs  and  external 
training  and  education  capabilities  (e.g.,  serious  games,  virtual  humans,  and 
simulations)  to  support  engaging  experiences  with  reduced  need  for  authoring 
(Sottilare  2015). 

Generalized  Intelligent  Framework  for  Tutoring  (GIFT)  (Sottilare  et  al.  2012a; 
Sottilare  et  al.  2013a):  An  open-source,  modular  architecture  whose  goals  are  to 
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reduce  the  cost  and  skill  required  for  authoring  adaptive  training  and  educational 
systems,  automate  instructional  delivery  and  management,  and  develop  and 
standardize  tools  for  the  evaluation  of  adaptive  training  and  educational 
technologies. 

Adaptive  training  and  education  research  at  ARL  are  being  conducted  across  6 
interdependent  research  vectors:  individual  learner  and  unit  modeling,  instructional 
management  principles,  domain  modeling,  authoring  tools  and  methods,  evaluation 
tools  and  methods,  and  architectural  and  ontological  support.  This  report  (1  of  6 
interdependent  research  outlines)  focuses  on  domain  modeling  research  for 
adaptive  training  systems  with  the  goal  of  guiding  learning  in  militarily  relevant 
training  and  educational  domains. 

Soldiers  operate  in  a  variety  of  complex,  dynamic,  ill-defined  domains  where  their 
ability  to  persevere  in  the  face  of  adversity,  adapt  to  their  situation,  collaborate,  and 
think  critically  are  key  to  the  successful  completion  of  their  assigned  missions.  To 
develop  and  exercise  these  skills,  it  is  paramount  for  Soldiers  to  train  in  challenging 
environments.  Currently  these  few  challenging  training  environments  have  been 
largely  provided  through  manpower-intensive  methods  or  systems  with  little  ability 
to  adapt  instruction  to  support  their  learning  needs.  To  illustrate  this  point,  Franke 
(2011)  asserts  that  through  the  use  of  case  study  examples,  instruction  can  provide 
the  pedagogical  foundation  for  decision  making  under  uncertainty.  However,  this 
approach  is  limited  in  implementation  by  the  expanse  of  potential  cases  that  would 
need  to  be  consistently  updated  and  maintained  to  support  large  populations  like 
the  US  Anny. 

As  noted  previously,  adaptive  systems  like  ITSs  have  been  shown  to  be  effective 
in  promoting  learning  in  primarily  static  (e.g.,  learners  seated  at  desktop  computers) 
instructional  settings  within  relatively  simple,  well-defined  domains  (e.g., 
mathematics  and  physics)  for  individual  learners.  For  our  purposes,  static 
instruction  includes  cognitive,  affective,  or  social  training  tasks  where  a  desktop 
computer  delivers  instruction  and  where  the  physical  movement  of  the  learner  is 
limited  to  activities  that  can  be  conducted  while  seated.  For  example,  static 
instruction  can  effectively  support  cognitive  tasks  involving  decision  making  and 
problem  solving  but  are  less  effective  for  training  tasks  involving  motion  and 
perception  (e.g.,  land  navigation  and  marksmanship).  Ideally,  we  desire  portable 
adaptive  instructional  capabilities  to  go  with  Soldiers  to  support  training  and 
education  at  their  point  of  need  across  a  wide  spectrum  of  Army  operational  tasks. 
Research  is  needed  to  develop  tools  and  methods  to  support  broader  domain 
modeling  that  is  representative  of  the  full  spectrum  of  Army  operational  tasks. 
Standards,  interoperability,  and  automation  (e.g.,  automated  scenario  generation) 
(Zook  et  al.  2012)  will  likely  play  a  significant  role  in  making  adaptive  training 
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practical.  In  this  way,  adaptive  training  technologies  will  have  the  greatest  impact 
on  organizational  learning  in  the  Army. 

4.  US  Army  Requirements  for  Adaptive  Training  Systems  and 
Instructional  Management  Research 

The  Army  science  and  technology  community  uses  Warfighter  Outcomes  (WFOs) 
as  the  authoritative  source  for  identifying  Warfighter  needs.  WFOs  are  used  to  share 
research  and  future  technology  solutions.  In  the  training  and  education  (TandE) 
domain,  the  adaptive  TandE  research  program  is  targeting  4  specific  requirements 
to  support  the  evolution  of  US  Army  training:  adaptive  training  and  education 
systems,  big  data,  training  at  the  point  of  need,  and  AI. 

4.1  Adaptive  Training  and  Education  Systems  and  Instructional 
Management 

The  primary  gap  to  be  addressed  under  this  Army  requirement  is  the  lack  of 
adaptive  systems  (e.g.,  intelligent  tutors)  to  support  individual  and  collective  (team 
or  unit)  training.  The  Army  needs  an  adaptive  training  and  education  capability  that 
is  persistent  and  easy  to  use/access  with  minimal  start-up  time.  There  are  also 
requirements  to  automate  an  informal  AAR  (also  known  as  postexercise  critique) 
to  reduce  the  time  and  skills  needed  to  produce  the  AAR  and  improve  its  focus  and 
quality.  Another  line  of  thought  also  notes  that  the  AI  in  ITSs  could  be  used  to 
facilitate  rapid  mission  planning  and  course-of- action  analyses  as  a  job  aid  in 
operational  contexts. 

An  effective  adaptive  training  capability  depends  on  sound  instructional 
management  practices.  Instructional  management  practices  consist  of  techniques, 
strategies,  and  tactics  applied  to  a  domain  of  instruction  to  optimize  performance 
outcomes  (Sottilare  et  al.  2014).  With  respect  to  GIFT,  this  vector  of  research  is 
concerned  with  identifying  instructional  best  practices  that  associate  with  all  facets 
of  learning  as  well  as  establishing  authoring  workflows  that  instantiate  those 
practices  across  multiple  environments  of  instruction.  The  goal  is  to  provide 
adaptive  training  solutions  that  are  sufficiently  tailored  for  each  individual  Soldier 
and  for  teams  of  Soldiers. 
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4.2  Big  Data  and  Instructional  Management 

The  primary  gap  to  be  addressed  under  this  Anny  requirement  is  that  there  is  a  lack 
of  capability  to  handle  and  process  large  amounts  of  structured  and  unstructured 
data  (also  referred  to  as  big  data).  One  capability  needed  is  a  structured  data 
analytics  program  linking  individual  data  (e.g.,  achievements)  to  required  long¬ 
term  competencies  in  military  occupational  specialties  (MOSs).  This  would  allow 
Soldiers  to  understand  where  they  rank  in  tenns  of  experiences  and  achievements 
among  other  Soldiers  in  their  MOS.  It  would  also  allow  the  Anny  to  identify 
specific  experiences  among  successful  Soldiers  in  that  MOS  and  provide  a  model 
for  other  Soldiers  in  that  MOS  to  follow.  This  data  could  also  be  used  by  course 
managers  and  instructors  to  continuously  improve  instruction  and  the  mental 
models  of  both  human  and  computer-based  instructors.  Finally,  data  collected  on 
trainee  learning  and  perfonnance  during  adaptive  training  experiences  could  be 
used  to  facilitate  unit  training  management  where  unit  commanders  would  have 
access  to  empirical  data  to  support  unit  training  decisions. 

A  goal  of  instructional  management  in  GIFT  is  to  optimize  perfonnance  and 
competency  outcomes  through  personalized  training  experiences  that  adapt  to  an 
individual’s  knowledge,  skills,  and  abilities  (KSAs).  A  connection  between  big  data 
and  instructional  management  is  applying  large  datasets  from  prior  course 
interactions  to  update  and  improve  technique  and  strategy  implementations  across 
all  available  courses.  A  challenge  in  defining  instructional  management  logic  in  a 
domain-independent  context  is  that  it  requires  generalizability  across  applications. 
An  issue  with  instructional  strategy-based  research  is  that  an  approach  taken  in  one 
domain  is  hard  to  translate  to  a  different  without  performing  extensive  research  to 
validate  its  application.  In  addition,  it  is  difficult  to  define  definitive  instructional 
management  logic  based  on  these  uncertainties.  As  such,  big  data  can  be  used  to 
account  for  this  uncertainty  by  applying  machine  learning  and  data  mining 
techniques  to  assess  specific  causal  relationships  between  instructional  practices 
and  outcomes  on  performance,  retention,  and  transfer.  This  application  of  big  data 
is  used  to  reinforce  instructional  management  models  by  optimizing  itself  over  time 
as  more  and  more  data  is  made  available. 

4.3  Training  at  the  Point  of  Need  and  Instructional 
Management 

The  primary  gap  to  be  addressed  under  this  Army  requirement  is  the  lack  of  an 
easily  accessible,  persistent,  cost-effective,  and  low-overhead  training 
environment.  A  capability  is  needed  to  bring  training  to  Soldiers  instead  of  Soldiers 
going  to  fixed  training  locations.  This  point-of-need  training  capability  would  be 
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easily  distributed,  web-based,  and  built  upon  open-enterprise  architecture  in  the 
cloud.  Army  training  and  educational  opportunities  would  be  available  on  demand 
anywhere  and  anytime.  However,  the  delivery  mechanism  (e.g.,  laptop  computer, 
mobile  device,  and  smart  glasses)  for  adaptive  training  is  critical  in  determining  the 
limitations  of  the  domain  model  scope  and  complexity.  For  example,  it  may  be 
extremely  difficult  to  train  all  the  complexities  of  a  psychomotor  task  in  a  desktop 
computer  setting. 

A  goal  of  instructional  management  in  GIFT  is  to  support  a  model  of  training  by 
convenience  and  to  have  mechanisms  to  support  this  form  of  education  in  the 
absence  of  live  instruction.  Effective  training  applications  administered  in  a  point- 
of-need  capacity  requires  4  primary  components  to  deliver  sound  instruction:  1)  the 
ability  to  monitor  trainee  interaction  and  behavior  to  accurately  assess  performance 
against  a  granular  concept  by  concept  model,  2)  the  ability  to  communicate 
guidance  and  feedback  messages  that  correspond  with  individual  performance,  3) 
the  ability  to  adapt  scenario/problem  elements  to  maintain  appropriate  challenge 
levels  for  promoting  flow,  and  4)  the  ability  to  manage  an  automated  AAR  to 
review  scenario  performance  and  promote  reflection  on  the  linkage  between 
outcomes  and  overall  learning  objectives.  Each  of  these  components  depend  on 
each  other  for  the  purpose  of  delivering  personalized  training  experiences  through 
an  easily  distributed,  web-based,  open-enterprise  architecture.  In  addition,  an 
optimal  adaptive  training  capability  will  leverage  existing  course  materials  to  better 
serve  the  development  of  a  competency  through  point-of-need  functions.  An 
example  is  providing  a  remedial  training  activity  on  an  identified  weakness 
between  training  events  to  better  prepare  that  individual  for  the  next  period  of 
instruction  (e.g.,  recognizing  trigger  squeeze  problems  in  the  Engagement  Skills 
Trainer  and  providing  a  remedial  multimedia  training  event  prior  to  the  next  round 
of  marksmanship  instruction  on  the  live  range). 

4.4  Artificial  Intelligence  Capabilities  and  Instructional 
Management 

The  primary  gap  to  be  addressed  under  this  Army  requirement  is  that  the  Anny 
lacks  an  automated  capability  to  replicate  the  complexity  and  uncertainty  of  the 
operational  environment.  This  gap  specifically  points  to  the  lack  of  adaptiveness  in 
virtual  humans,  intelligent  tutoring  systems,  and  other  training  capabilities.  This 
gap  leads  to  Soldiers  developing  training  response  strategies  that  result  in  less- 
challenging  training  over  time  along  with  lower  engagement  and  lower  levels  of 
learning  and  transfer  of  skills  to  more  challenging  operational  environments. 
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A  goal  of  instructional  management  in  GIFT  is  to  facilitate  robust  AI  capabilities 
that  enhance  the  realism  and  playability  of  simulation-based  training  exercises, 
specifically  those  that  utilize  virtual  humans  and  semi-automated  forces.  These 
elements  must  be  embedded  with  AI  capabilities  that  provide  an  automated  reactive 
capacity  to  trainee  inputs  and  actions  that  adhere  to  the  complexity  and  uncertainty 
of  an  operational  environment.  In  terms  of  virtual  humans,  these  entities  must  have 
logic  that  supports  realistic  movements  and  communication  exchanges  that  reflect 
back  to  a  culture  or  operational  environment.  This  requires  AI  embedded  within 
virtual  humans  that  accounts  for  cultural  norms  and  customs,  along  with  the  ability 
for  the  entity  to  adapt  its  behavior  based  on  cues  and  actions  perceived  from  the 
trainee  (rolling  of  the  eyes,  change  in  vocal  intonation,  failing  to  account  for 
appropriate  cultural  greeting,  etc.).  AI  also  needs  to  be  embedded  in  virtual  humans 
that  enables  their  use  as  virtual  teammates  in  a  collaborative  training  scenario. 

This  will  allow  for  effective  training  of  team-oriented  missions  without  the 
requirement  of  using  an  all-human  team.  In  terms  of  semi-automated  forces,  which 
are  commonly  used  in  tactical  training  events  within  environments  such  as  Virtual 
Battlespace  3  (VBS3),  AI  techniques  must  be  established  that  allow  a  group  of 
forces  to  adapt  their  movements  over  time,  as  it  can  autonomously  learn  from 
actions  and  tactics  executed  by  a  trainee  or  team  of  trainees.  This  will  allow  a  set 
of  enemy  forces  to  better  adapt  itself,  much  like  in  the  real  world,  for  the  purpose 
of  creating  richer  training  experiences  that  increase  complexity.  In  addition,  AI 
techniques  must  be  investigated  to  ease  the  development  of  training  scenarios  to 
avoid  issues  of  “replay-ability”.  A  system  such  as  VBS3  would  benefit  from 
technologies  that  enable  the  generation  of  multiple  scenarios  for  training  purposes 
based  solely  around  a  defined  set  of  tasks,  conditions,  and  standards.  This  promotes 
better  training  because  a  trainee  needs  to  adapt  his/her  application  of  knowledge 
and  skills  to  an  unknown  event  rather  than  gaming  a  scenario  by  learning  the 
various  cues  and  scripts  executed  by  an  enemy  entity  (i.e.,  knowing  an  insurgent  is 
hiding  behind  a  specific  door  in  a  specific  hallway). 

5.  Understanding  the  Dimensions  of  Instructional 
Management 

There  are  4  typical  elements  that  comprise  ITSs,  a  prime  example  of  an  adaptive 
training  and  education  system:  a  learner  or  trainee  model,  an  instructional  or 
pedagogical  model,  a  domain  model,  and  some  type  of  user  interface.  The  domain 
model  typically  includes  an  expert  or  ideal  student  model  by  which  the  adaptive 
system  measures/compares/contrasts  the  progress  of  the  learner  toward  learning 
objectives.  The  domain  model  also  includes  the  training  environment,  the  training 
task,  and  all  of  the  associated  instructional  actions  (e.g.,  feedback,  questions,  hints, 
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pumps,  and  prompts)  that  could  possibly  be  delivered  by  the  adaptive  system  for 
that  particular  training  domain.  Typical  interaction  between  the  learner,  the  training 
environment,  and  the  adaptive  system  (tutoring  agents)  is  shown  in  Fig.  1 . 
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Fig.  1  Adaptive  training  interaction 

Typical  training  systems  examine  the  interaction  between  the  learner  and  the 
training  environment  to  measure  progress  toward  learning  objectives.  The  learner 
acts  on  the  environment  (e.g.,  opens  a  door  or  makes  a  choice  to  move  into  the  room 
or  stay  outside)  and  then  observes  any  changes  or  reactions  within  the  environment. 
Adaptive  systems  add  a  layer  of  software-based  tutoring  agents  that  are  designed 
to  guide  the  learner  in  much  the  same  way  as  a  human  tutor  interacts  with  a  learner. 
The  tutoring  agents  observe  the  behaviors  of  the  learner  to  assess  their  states  (e.g., 
performance  and  attitudes)  and  interact  with  the  learner  to  provide  support, 
direction,  and  instruction.  In  addition,  they  track  the  effect  of  interactions  on 
learning.  Tutoring  agents  also  interact  with  the  training  environment  and  may 
manipulate  the  environment  to  present  more-  or  less-challenging  scenarios  in 
response  to  the  assessed  state  of  the  learner. 

5.1  Theoretical  Foundations  of  Instructional  Management 

Instructional  strategies  have  been  advocated  by  researchers  and  practitioners  in 
many  different  fields,  including  education,  educational  psychology,  cognitive  and 
learning  sciences,  military  training,  computer-based  training,  AI  in  education, 
computer-supported  collaborative  learning,  educational  data  mining,  etc.  (Sottilare 
et.  al.  2014).  While  these  fields  often  vary  across  intent  and  scope  of  research, 
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common  themes  have  been  recognized  across  multiple  reports  involving 
interdisciplinary  research  groups  organized  by  the  government  and  research 
organizations.  Examples  include  (as  highlighted  in  Sottilare  et.  al.  2014)  the 
following: 

•  A  Roadmap  to  Educational  Technology.  National  Science  Foundation, 
Arlington,  VA;  2010.  http://www.cra.org/ccc/docs/groe/GROE  Roadmap 
for  Education  Technology  Final  Report.pdf. 

•  The  Army  Learning  Concept  for  2015.  US  Army,  Washington,  DC;  2011. 
http://www-tradoc.army.mil/tpubs/pams/tp525-8-2.pdf. 

•  Committee  on  Science  Learning:  Computer  Games,  Simulations,  and 
Education.  National  Academy  of  Sciences,  Washington,  DC;  2011. 
http://www.nap.edu/catalog.php?record_id=13078. 

•  Assessing  21st  Century  Skills.  National  Academy  of  Sciences,  Washington, 
DC;  2011.  http://www.nap. edu/catalog.php?record_id=13215#toc. 

•  Improving  Adult  Literacy  Instruction.  National  Academy  of  Sciences, 
Washington,  DC;  2012.  http://www.nap.edu/catalog.php7record_kU1342. 

•  Organizing  Instruction  and  Study  to  Improve  Student  Learning.  Institute  of 
Education  Sciences  of  the  United  States  Department  of  Education, 
Washington,  DC;  2007.  http://ies.ed.gov/ncee/wwc/pdf/practice_guides 
/20072004.pdf. 

•  Lifelong  Learning  at  Work  and  at  Home.  American  Psychological 
Association  and  Association  for  Psychological  Sciences,  Washington,  DC; 
2007.  See  Graesser  AC.  Inaugural  editorial  for  Journal  of  Educational 
Psychology.  2009;  1 0 1  (2):259-26 1 . 

These  reports  emphasize  instructional  strategies  that  are  supported  by  empirical 
tests  with  scientific  methodologies;  therefore,  the  strategies  are  grounded  in  science 
and  are  evidence-based.  It  is  important  that  many  of  the  strategies  recommended 
from  these  groups  are  geared  toward  live  instruction. 

In  terms  of  implementation,  instructional  management  involves  both  an  adaptive 
outer  loop  (i.e.,  adjusting  problem  selection  based  on  perfonnance  outcomes)  and 
an  adaptive  inner  loop  (i.e.,  providing  real-time  error-sensitive  feedback  or 
manipulating  the  scenario  directly)  function  that  accounts  for  individual  differences 
across  KSAs  within  a  given  domain.  An  adaptive,  intelligent  learning  environment 
needs  to  launch  the  right  instructional  strategies  at  the  right  time  in  a  mechanism 
that  attempts  to  be  sensitive  to  the  learner  model,  maximize  learning  and 
motivation,  and  minimize  training  time  and  costs. 
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5.2  Instructional  Management  in  GIFT 


Before  we  examine  specific  goals  and  research  interests  associated  with 
instructional  management  in  GIFT,  it  is  important  to  review  some  high-level 
components  of  the  architecture.  This  involves  an  understanding  of  how  information 
and  data  is  represented  and  how  these  representations  ultimately  inform  the  updated 
Learning  Effect  Model  (LEM)  (Sottilare  2015  in  press;  Sottilare  et  al.  2015,  Fig. 
2).  The  important  takeaway  of  the  LEM  is  the  flow  of  data  with  respect  to  the 
selection  of  an  instructional  practice.  The  effect  chain  is  influenced  by  both 
historical  data  (e.g.,  prior  experience,  prior  knowledge,  and  learner  traits) 
maintained  over  time  and  real-time  data  captured  during  a  specific  interaction.  This 
data  is  used  to  adapt  instruction  on  2  facets:  1)  an  inner-loop  capacity  using  data  to 
influence  interaction  within  a  single  problem  or  scenario  by  providing  guidance  or 
adjusting  difficulty  levels  similarly  to  Vygotsky’s  (1978)  zone  of  proximal 
development  (ZPD)  and  2)  an  outer-loop  capacity  that  configures  the  next  event 
experienced  by  a  learner  based  on  assessments  from  a  prior  event  or  through 
predictions  based  on  historical  representations  and  learner  traits  (VanLehn  2006). 
This  might  involve  selecting  a  new  problem/scenario,  managing  a  remediation 
event,  moving  on  to  a  new  section  of  the  course,  administering  an  AAR,  or  ending 
the  course. 
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Fig.  2  Updated  individual  learning  effect  model  (Sottilare  2015  in  press;  Sottilare  et  al. 

2015) 

In  terms  of  the  inner-loop  adaptive  function,  Assistive  Technology  Learning 
through  a  Unified  Curriculum  (ATLEC)  primarily  interacts  on  learner  assessments 
occurring  in  real  time  for  both  performance  and  affective  related  states.  Ultimately, 
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raw  interaction  data  are  used  to  infer  a  learner  state.  This  involves  robust 
assessment  techniques  that  can  accurately  gauge  an  individual  learner’s 
perfonnance  for  a  given  task,  the  affective  responses  they  are  having  within  that 
task,  and  their  estimated  competency  for  the  domain  that  task  is  designed  to  train. 
This  inferred  learner  state  is  used  to  inform  the  selection  of  an  instructional  strategy 
to  mediate  the  learning  experience.  In  the  current  baseline  of  GIFT  (GIFT  2015), 
the  instructional  strategies  supported  for  the  inner  loop  consists  of  “provide 
guidance”,  “adapt  scenario”,  “administer  assessment”,  and  “do  nothing”.  These 
strategies  are  represented  as  high-level  domain-independent  descriptors  of  an 
action  the  system  can  take  within  a  given  learner  event.  These  high-level  actions 
must  then  be  translated  into  a  specific  tactic  of  execution  (see  Sottilare  et.  al.  2014 
for  a  comprehensive  breakdown  of  strategies  versus  tactics). 

For  outer-loop  adaptive  functions,  components  of  the  ATLEC  are  administered  up 
front  to  configure  a  lesson’s  sequence  of  interaction.  Learner  data  inform  strategy 
selections  that  associate  with  Merrill’s  (1994)  Component  Display  Theory.  This 
interaction  is  currently  encapsulated  in  a  tool  used  by  GIFT  called  the  Engine  for 
Management  of  Adaptive  Pedagogy  (EMAP)  (see  Fig.  3).  In  the  latest  GIFT 
baseline,  the  EMAP  is  used  to  guide  a  learner  through  a  set  of  interactions  that 
focuses  on  1)  presentation  of  rules  for  a  domain  or  task,  2)  presentation  of  examples 
where  those  rules  are  being  applied,  3)  administering  of  a  knowledge  assessment 
that  gauges  a  learner’s  ability  to  recall  facts,  and  4)  administering  of  a  practice 
assessment  that  gauges  a  learner’s  skill  for  performing  tasks  associated  with  the 
domain  of  instruction  (Goldberg  et  al.  2012;  Wang-Costello  et.  al.  2013). 
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Fig.  3  GIFT’S  EMAP  (Goldberg  and  Hoffman  2015) 

With  a  current  layout  of  the  functions  GIFT  supports  from  an  instructional 
management  perspective,  the  remainder  of  the  research  outline  will  target  specific 
utilities  of  interest  that  a  domain- independent  architecture  needs  for  wide  use  across 
military,  academic,  and  industry  settings.  Specific  goals  of  desired  functions  will 
be  presented,  followed  by  recognized  subvectors  of  interest  that  require  substantial 
research  to  determine  their  optimal  conditions  for  use  in  a  learning  event.  As  GIFT 
is  intended  to  be  a  unified  standard  for  building  ITSs  of  all  kinds,  it  is  important  to 
recognize  the  varying  instructional  management  practices  that  can  be  implemented 
and  what  conditions  are  best  for  their  use. 
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6.  Instructional  Management  Research  Goals  and  Challenges 


The  foundational  goal  of  adaptive  training  research  at  the  ARL  is  to  model  the 
perception,  judgment,  and  behaviors  of  expert  human  tutors  to  support  practical, 
effective,  and  affordable  learning  experiences  guided  by  computer-based  agents. 
To  this  end,  5  primary  challenges  in  instructional  management  for  adaptive  training 
systems  have  been  identified  and  are  discussed  in  this  section. 

In  the  following  subsections,  we  review  the  specific  overarching  themes  associated 
with  instructional  management  research  in  GIFT.  Each  theme  is  represented  as  a 
subvector  in  this  research  outline.  While  reading  through  these  research  thrusts, 
make  sure  to  conceptualize  their  application  across  various  training  environments 
and  use  cases.  While  the  subvectors  can  be  somewhat  confined  in  terms  of 
instructional  management  intent,  options  for  how  they  can  be  applied  across  all 
types  of  training  environments  are  vast.  This  includes  considering  individualized 
versus  team-based  training  events  and  the  technologies  being  applied  to  facilitate 
the  training  itself. 

6.1  Guidance  and  Scaffolding 

Scaffolding  is  a  term  used  to  describe  the  application  of  instructional  supports  to 
assist  a  learner  in  developing  knowledge  and  skills  that  the  learner  would  not 
achieve  when  left  to  their  own  devices.  Holden  and  Sinatra  (2014)  define  scaffolds 
as  a  temporary  application  of  strategy  that  is  gradually  removed  (“faded”)  once  a 
learner  demonstrates  an  increase  in  proficiency  or  skill.  The  word  ‘strategy’  is  used 
loosely  here,  as  it  can  be  a  number  of  interventions  that  range  from  guidance, 
feedback,  scenario/problem  manipulation,  and  remediation.  Prior  to  the  advent  of 
computer-based  training  environments  and  ITSs,  an  expert  adult  or  peer  (parent, 
teacher,  classmate,  teammate,  etc.)  provided  scaffolding  practices.  This  individual 
acted  as  an  expert  facilitator  of  KS  As  required  for  learning,  motivator,  model,  and 
means  for  the  learner  to  reflect  (Puntambekar  and  Hubscher  2005).  As  such,  much 
of  the  applied  guidance  and  scaffolding  practices  in  adaptive  training  environments 
are  based  on  what  effective  instructors  and  tutors  do  in  real  life.  In  examining 
models  of  expert  human  tutors,  the  following  themes  have  been  identified  for 
effective  interaction: 

•  Demonstrate  credible  knowledge  of  the  domain  under  training  (e.g.,  tactical 
combat  casualty  care). 

•  Read  cues  from  the  learner  and  adapt  instruction  in  real  time  to  meet  their 
changing  needs. 
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•  Encourage  question  asking. 

•  Provide  indirect  feedback. 

•  Assess  learning  often. 

When  designing  scaffolds  and  the  logic  associated  with  their  execution,  3 
dimensions  require  consideration:  what  to  scaffold,  when  to  scaffold,  and  how  to 
scaffold  (Azevedo  and  Jacobson  2008).  With  respect  to  GIFT,  each  of  these 
dimensions  must  be  conceptualized  in  a  domain-agnostic  fashion.  The  first 
consideration  that  will  dictate  how  to  proceed  is  based  on  available  tools  and 
methods  for  scaffolding.  In  this  vein,  the  mode  of  interaction  and  the  specific 
training  application  itself  will  detennine  what  approaches  can  be  implemented 
(Goldberg  and  Cannon-Bowers  2015).  For  instance,  interacting  in  VBS3  affords 
different  scaffolding  strategies  when  compared  with  interacting  with  a  negotiation 
trainer  on  a  tablet.  The  type  of  state  (e.g.,  performance,  affective)  information  being 
monitored  in  the  learner  model  (Shute  et  al.  2013),  the  type  of  communication 
interfaces  available  for  presenting  information  (e.g.,  GIFT’S  Tutor  User  Interface, 
smartphone,  smart  glasses,  haptic  device),  and  the  type  of  adaptations  supported  by 
the  training  application  (e.g.,  changing  the  weather  in  VBS3  to  increase  the 
complexity  of  scenario)  all  impact  the  types  of  scaffolds  that  can  be  built  to  support 
a  specific  course  or  lesson. 

Once  one  recognizes  the  available  tools  to  trigger  and  support  scaffolding  practices, 
detennining  what  to  scaffold  must  be  addressed  next.  What  to  scaffold  can  be 
represented  in  a  generalized  fashion  and  is  based  on  defined  learning  objectives  and 
barriers  to  a  learner  perfonning  successfully.  This  could  include  scaffolding 
specifically  on  the  cognitive  level  that  accounts  for  the  domain  or  the  task 
procedures  themselves.  Or  scaffolds  can  account  for  different  constructs  associated 
with  the  learning  process,  such  as  metacognition  (i.e.,  to  regulate  goal  planning, 
perfonnance  monitoring,  and  help-seeking)  and  affect  (i.e.,  to  regulate  motivation, 
boredom,  frustration,  and  confusion)  (van  de  Pol  et  al.  2010).  Recognizing  what 
one  wants  to  scaffold  along  with  what  tools  and  methods  one  has  to  support  those 
types  of  interventions  will  lend  itself  into  building  the  specific  scaffolds  for 
implementation. 

With  scaffolds  conceptually  established,  the  next  step  is  building  logic  for 
detennining  when  to  apply  a  scaffold  and  how  to  appropriately  fade/adapt  its  use 
to  promote  efficient  transfer  of  execution.  A  common  perspective  to  account  for 
this  challenge  is  based  on  Vygotsky’s  (1978)  ZPD.  ZPD  is  an  optimal  and  efficient 
zone  of  learning  that  elevates  the  student  from  his/her  current  and  actual 
developmental  level  to  one  of  more  potential  through  balancing  challenge  with 
ability  and  through  fonnative  guidance  to  enhance  skill  (Vygotsky  1978).  In  its 
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simplest  form,  the  ZPD  creates  a  formalized  state  space  representation  that  enables 
a  system  to  contextualize  what  a  learner  is  experiencing  for  a  given  training  domain. 
For  example,  if  a  training  system  launches  a  scenario  with  a  precon  ligured 
difficulty  setting  of  expert  and  a  trainee’s  skill  state  is  defined  as  journeyman,  the 
system  may  trigger  specific  scaffolds  that  are  intended  to  support  the  maturation  of 
skill  levels  to  meet  the  challenge  level  of  a  selected  scenario.  In  this  instance,  if 
perfonnance  assessments  associate  execution  with  increased  skill  levels,  the  system 
must  be  able  to  recognize  this  shift  and  fade  scaffolds.  From  the  opposite  end,  if  a 
learner  demonstrates  skills  that  fall  below  his  or  her  predicted  ability  levels,  the 
system  must  be  able  to  appropriately  trigger  scaffolds  to  compensate  for  that 
inaccurate  upfront  classification.  This  must  be  possible  for  both  dynamic  scenario- 
driven  tasks  as  well  as  for  training  applications  that  use  discrete  problem  sets  that 
allow  adaptation  between  each  problem. 

Much  of  the  described  scaffolding-  and  guidance-focused  end-state  objectives  are 
composed  of  a  mix  of  inner-loop  and  outer-loop  pedagogical  practices.  In  short,  the 
training  application  being  used  will  dictate  the  type  of  scaffolding  that  can  be 
executed.  While  GIFT  is  domain-agnostic,  once  a  training  application  is  applied, 
the  next  step  is  conceptualizing  what  adaptive  functions  are  supported  and  what 
authoring  processes  are  required.  In  an  ideal  end-state,  a  constraint-based, 
intelligent  pedagogical  agent  can  be  applied  to  guide  the  authoring  processes  and 
to  manage  run-time  decisions.  These  decisions  center  on  selecting  appropriate 
domain  independent  strategies,  translating  strategies  into  optimal  tactics  (actions) 
based  on  situational  context,  and  then  presenting  the  tactic  (take  action)  through  the 
delivery  of  feedback,  providing  direction,  or  changing  the  training  environment. 

When  considering  scaffolding  for  adaptive  training  environments,  several  needed 
areas  of  research  emerge.  These  areas  are  relevant  due  to  the  advancing  popularity 
in  ITSs  and  the  relative  infancy  of  these  environments  applied  in  military  contexts. 
Holden  and  Sinatra  (2014)  identify  several  necessary  future  research  areas 
involving  scaffolding  and  GIFT.  One  involves  defining  the  different  focuses  to 
scaffolding  and  distinguishing  the  conditions  under  which  its  implementation  is 
best.  Several  specific  types  of  scaffolding  exist  on  a  theoretical  level,  including 
conceptual,  metacognitive,  procedural,  and  strategic  (Hannafin  et  al.  1999).  As 
such,  the  word  ‘scaffold’  applied  in  this  context  is  as  a  very  broad  term  that 
associates  with  multiple  forms  of  transitions,  interventions,  and  adaptations 
performed  by  a  system.  In  tenns  of  pedagogical  intent,  a  range  of  scaffolds  can  be 
applied  to  associate  with  the  following: 

•  Error-Sensitive  Feedback:  An  intervention  triggered  when  the  learner 
commits  errors  that  are  either  individually  or  cumulatively  significantly 
divergent  from  the  ideal  as  defined  in  the  expert  model  of  the  ITS. 
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•  Mastery  Learning:  A  strategy  where  the  ITS  ensures  that  the  learner 
masters  (can  recall  and  apply)  prerequisite  lessons  or  concepts  before 
allowing  the  learner  to  move  on  to  the  next  lesson/concept. 

•  Adaptive  Spacing  and  Repetition:  A  strategy  where  the  learner  more 
easily  recalls  knowledge  items/objects  when  the  knowledge  is  exposed  to 
the  learner  repeatedly  over  a  long  time-span  rather  than  repeatedly  studied 
during  a  short  span  of  time  (Benjamin  and  Tullis  2010;  Dempster  1989). 

•  Metacognitive  Prompting:  A  strategy  where  the  ITS  encourages  the 
learner  to  self-reflect  and  evaluate,  self-explain,  and  self-correct  rather  than 
provide  the  answer  directly. 

•  Fading  Worked  Examples:  “[A]  step-by-step  demonstration  of  how  to 
perfonn  a  task  or  how  to  solve  a  problem”  from  which  parts  have  been 
deliberately  removed  or  faded  (Clark  et  al.  2006,  p.  190). 

While  scaffolds  are  very  domain-relevant  and  application-dependent,  tools  and 
methods  must  be  established  in  GIFT  to  support  building  and  implementing  optimal 
scaffolds  across  all  types  of  supporting  training  environments.  This  also  extends 
into  investigating  scaffolds  for  team-based  training  events.  From  this  association, 
the  following  potential  research  questions  arise: 

•  What  is  the  best  approach  to  the  authoring  run-time  instructional 
management  logic  in  GIFT? 

•  Are  there  pedagogical  best-practice  heuristics  that  generalize  across  training 
environments  with  varying  levels  of  interactivity  and  user  control? 

•  How  do  scaffolding  best  practices  translate  to  collaborative  and  team-based 
training  exercises? 

•  What  are  the  variations  in  feedback  considerations  when  comparing 
individualized  instruction  with  team-based  training? 

•  What  role  do  worked  examples  play  in  military  relevant  training 
environments,  and  how  can  SMEs  use  this  pedagogical  function  to  create 
better  training? 

•  What  probabilistic  modeling  techniques  account  for  system  uncertainties 
and  optimize  instructional  management  practices  and  tutorial  planning  logic 
over  time  to  avoid  solely  deterministic  execution? 

•  What  architectural  modifications  are  required  to  support  contextualized 
feedback  prompts  that  adhere  to  a  domain-agnostic  pedagogical  structure 
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(i.e.,  communicating  scenario  relevant  feedback  by  including  interaction 
data  available  in  log  files)? 

•  How  much  control  should  the  learner  have  on  guidance  and  scaffolding 
support  (i.e.,  help  seeking)? 

•  When  should  guidance  and  scaffolding  be  on  demand  (e.g.,  learner  requests 
hint),  triggered  by  a  student/system  event  or  state,  or  blended? 

•  What  perfonnance  factors  and  individual  differences  tracked  in  the  learner 
model  should  moderate  scaffold  selection  and  fading  logic? 

•  How  do  generalized  scaffolding  techniques  vary  across  training  delivery 
modalities  (e.g.,  laptop,  smartphone/tablet,  simulators,  and  smart 
glasses/head-mounted  displays)? 

•  How  can  SMEs  embed  scaffold  supports  in  virtual  world  game 
environments  commonly  used  for  training  purposes  (e.g.,  VBS3)? 

•  Under  what  conditions  does  an  adaptive  training  environment  provide 
guidance  versus  adapting  the  scenario  or  problem  selection? 

•  What  assessment  types  trigger  variations  in  scaffolding  techniques  (e.g., 
conceptual  versus  affective  versus  procedural  versus  metacognitive)? 

6.2  Social  Dynamics  and  Virtual  Humans 

In  this  research  subvector,  we  review  end-state  goals  associated  with  the  role  of 
social  dynamics  and  virtual  humans  in  managing  instruction  within  adaptive 
training  environments.  This  line  of  research  associates  with  tenets  of  Social 
Cognitive  Theory  in  that  learning  is  theorized  to  be  an  inherently  social  process 
(Bandura  1986;  Vygotsky  1978).  As  such,  techniques  should  be  applied  to  account 
for  high-valued  social  elements  that  can  potentially  improve  a  piece  of  educational 
technology.  In  addition,  Army  task  domains  can  involve  highly  socialized 
interactions.  To  support  these  task  characteristics,  adaptive  training  systems  should 
account  for  the  types  of  interactions  a  Soldier  might  face  (e.g.,  negotiating  with  a 
village  elder)  and  the  variables  that  may  influence  their  course  of  action  (e.g., 
cultural  norms  for  greetings  and  negotiations).  From  an  adaptive  training 
perspective,  social  dynamics  and  virtual  human  research  is  focused  on  the 
following: 

•  Using  technology  and  AI  to  replicate  interactive  discourse  common  in 
educational  and  operational  settings. 
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•  Using  technology  to  embed  social  elements  in  the  environment,  such  as 
virtual  humans,  to  create  a  social  grounding  function  for  delivering 
information/guidance. 

•  Using  technology  and  AI  to  create  realistic  and  reactive  virtual  humans  as 
training  elements  in  a  simulation  or  scenario  (e.g.,  role  player  and 
teammate). 

•  Using  technology  to  create  a  social  forum  for  the  purpose  of  supporting 
peer-to-peer  and  collaborative  learning,  both  from  a  real-time  perspective 
and  a  time-agnostic  approach  allowing  interaction  at  convenience. 

Each  referenced  focus  area  is  based  on  a  specific  social  element  of  interest  to  the 
GIFT  community.  Of  importance  is  the  application  of  these  elements  within  the 
standardized  architecture  inherent  to  GIFT.  Specifically,  how  can  the  tools  and 
methods  built  to  afford  these  capabilities  be  translated  to  support  ease  of  application 
within  any  type  of  training  event  and  within  any  type  of  training  environment 
supported  by  GIFT?  Each  identified  element  associates  distinctly  different  research 
questions  and  associates  distinctly  different  scientific  disciplines.  It  is  a  very  broad 
subvector  of  instructional  management  with  many  dependencies  to  other  elements 
of  the  GIFT  research  program. 

In  the  case  of  using  ITS  technologies  to  replicate  interactive  discourse,  an  end-state 
goal  is  to  establish  state-of-the-art  natural  language  processes  to  create  a  robust  tool 
capable  of  dynamic  question  and  answer  (QandA)  exchanges.  In  this  instance,  we 
want  a  training  environment  to  be  able  to  push  a  question  to  a  learner  and  a  trainee 
to  be  able  to  ask  questions  of  the  ITS,  ideally  through  natural  language  and  open 
response  input  methods.  As  highlighted  previously,  a  characteristic  of  an  effective 
tutor  is  one  who  encourages  a  learner  to  ask  questions.  This  process  in  itself 
promotes  learning  through  abstraction  and  reflection.  As  such,  we  need  a  discourse 
capability  that  can  accurately  interpret  user  responses  and  intent  as  they  relate  to 
the  semantic  space  of  instruction,  and  we  want  this  capability  to  associate  with  pre¬ 
existing  training  applications  (i.e.,  natural  discourse  QandA  in  parallel  with 
executing  a  VBS3  scenario).  This  capability  can  be  applied  in  multiple  training 
instances  and  under  many  conditions,  including  1)  discourse  to  support  reflective 
QandA  sessions  to  promote  higher  order  cognitive  thinking,  2)  discourse  to  support 
training  events  that  involve  social  exchanges  to  meet  certain  negotiation  objectives, 
and  3)  discourse  to  support  realistic  communication  with  virtual  entities  in  an 
environment  that  associate  with  both  friendly  and  opposing  forces.  Much  like 
scaffolding  capabilities,  the  application  of  natural  language  discourse  in  an  adaptive 
training  event  associates  many  dependencies  across  the  various  research  vectors. 
Specifically,  domain  modeling,  authoring  processes,  and  architecture  requirements 


22 


are  the  greatest  considerations  when  it  comes  to  implementing  this  approach  to 
instructional  management. 

Next,  virtual  humans  are  identified  as  key  technology  pieces  in  extending  adaptive 
training  experiences  to  account  for  varying  roles  in  the  learning  process.  From  an 
ITS  support  perspective,  virtual  human  research  is  focused  on  the  application  of 
technology  to  provide  an  interactive  communication  layer  that  grounds  all  system¬ 
generated  prompts  with  a  social  element.  An  overarching  intent  is  to  facilitate 
interaction  and  communication  with  a  computer  in  a  way  that  is  natural  and 
realistic.  The  goal  is  to  support  highly  engaging  and  interactive  experiences  through 
socialized  sequencing  of  interaction  and  enhancing  system  communication  by 
interfacing  with  a  learner  through  comfortable  modalities.  Much  of  the  prior 
research  in  this  area  focuses  on  the  trust  and  perception  of  technology  in  facilitating 
a  role  traditionally  managed  by  a  person  and  the  impact  on  motivation  and  effort 
(Holden  and  Goldberg  2011;  Kim  and  Baylor  2006;  Veletsianos  2010;  Veletsianos 
et  al.  2009). 

Virtual  humans  can  also  facilitate  critical  role  players  in  training  events.  In  these 
instances,  AI  methods  allow  virtual  entities  to  realistically  react  to  environmental 
stimuli  and  user  inputs  in  a  nonscripted  fashion,  making  the  experience  more 
natural  and  engaging.  Social  media  technologies  are  believed  to  offer  innovative 
tools  for  instructional  management  practices  that  have  yet  to  be  fully  taken 
advantage  of.  As  a  result,  research  is  required  to  better  understand  how  best  those 
tools  and  methods  can  be  applied.  This  involves  the  use  of  social  media  as  a  means 
to  collect  valuable  user  input  on  training  materials  to  determine  objects  that  need 
modifications,  as  well  as  investigating  how  social  media  can  be  used  to  promote 
collaborative  learning,  competition,  and  distributed  guidance. 

6.3  Metacognition  and  Self-Regulated  Learning 

SRL  theory  describes  the  process  of  taking  control  of  and  evaluating  one’s  own 
learning  and  behavior  (Butler  et  al.  2011).  As  a  higher-order  cognitive  function, 
SRL  is  guided  by  metacognitive  processes  (i.e.,  the  knowledge  and  regulation  of 
one’s  own  cognition),  strategic  actions  and  behaviors  (i.e.,  planning,  monitoring, 
and  assessing  one’s  own  perfonnance),  and  motivational  components  (i.e.,  goal 
setting  and  self-efficacy)  (Flavell  et  al.  1985;  Schraw  et  al.  2006).  These  functions 
allow  self-regulated  learners  to  set  goals,  monitor  their  progress  toward  defined 
goals,  and  adapt  and  regulate  their  cognition,  motivation,  and  behavior  to  reach  the 
specified  goals  (Anderman  and  Corno  2013;  Bransford  et  al.  2000).  These 
characteristics  also  associate  with  desired  competencies  within  the  ALM  that 
adaptive  training  solutions  are  intended  to  instill.  As  such,  research  in  the 
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instructional  management  vector  is  focused  on  the  application  of  models  and 
strategies  for  enhancing  metacognitive  awareness  and  regulation. 

This  approach  to  instructional  management  varies  from  traditional  guidance  and 
scaffolding  techniques,  as  it  focuses  on  behavior  and  application  of  strategy,  rather 
than  on  task-dependent  perfonnance.  One  such  question  is  based  on  GIFT 
supporting  SRL  and  the  efficacy  of  defining  persistent  metacognitive  strategies  that 
can  be  applied  across  domain  applications.  Currently,  GIFT  pedagogy  is  heavily 
focused  on  error-sensitive  feedback.  It  works  with  system  authors  by  translating 
instructional  strategy  recommendations  communicated  by  GIFT’S  pedagogical 
module  into  tactics  as  they  relate  to  the  specific  training  context.  These  tactics  are 
used  during  ITS  runtime  and  selected  based  on  a  learner’s  individual  differences. 
Currently,  feedback  in  GIFT  is  domain-dependent  and  requires  explicit  content 
linked  to  each  concept  modeled.  When  it  comes  to  metacognitive  feedback,  what 
are  the  implications  to  a  domain-independent  approach?  First,  modeling  techniques 
need  to  be  developed  to  monitor  an  individual’s  practice  of  metacognitive  strategies 
that  can  be  expressed  in  a  generalized  format.  An  example  would  be  incorporating 
a  combined  modeling  approach,  as  described  in  Biswas  et  al.  (2014),  or  by  adapting 
a  help-seeking  model,  as  highlighted  in  Koedinger  et  al.  (2009). 

One  such  approach  is  researching  and  establishing  models  based  around  commonly 
available  GIFT  interactions  (e.g.,  request  hint  button).  How  can  we  use  these 
available  data  inputs  to  build  a  representation  of  how  effective  students  use  the 
interface  to  solve  problems  and  troubleshoot  errors?  This  approach  can  aid  in 
detecting  learners  not  practicing  good  metacognitive  behaviors  through  machine 
learning  and  data  mining  practices  and  can  be  used  to  trigger  feedback  interventions 
to  improve  their  understanding  of  available  strategies.  With  modeling  techniques 
in  place,  generic  strategies  and  tactics  can  be  identified  that  are  based  on  effective 
metacognitive  behavior.  In  this  instance,  the  generic  strategy  of  “provide  guidance” 
can  be  linked  with  a  generic  tactic  of  “you  are  ignoring  available  resources”,  thus 
preventing  any  explicit  authoring  from  a  system  developer.  While  tactics  can  be 
represented  in  a  domain-independent  fonnat,  their  effect  is  relatively  unknown. 

Regardless  of  the  instructional  management  intent,  just  like  all  other  facets  of 
instruction,  metacognitive  tutoring  depends  on  sound  assessment  practices  that 
ultimately  inform  the  type  of  intervention  to  execute.  Due  to  difficulties  in  inferring 
upon  interaction  data  alone,  another  area  of  interest  is  the  application  of  alternative 
assessment  techniques  to  better  gauge  an  individual’s  understanding  of  the  task  and 
the  resources  available  to  meet  objectives.  This  involves  inference  procedures  on 
interaction  patterns  and  existing  perfonnance  outcomes,  as  well  as  devising 
approaches  to  collect  new  information  that  is  not  implicitly  available  in  the  training 
environment. 
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Beyond  all  of  the  associated  types  of  scaffolds  that  can  serve  metacognitive 
development,  how  does  one  implement  such  practices?  A  challenge  that  must  be 
addressed  is  establishing  an  authoring  environment  and  workflow  for  supporting 
SRL-derived  modeling  techniques  and  linking  outputs  to  prescribed  strategies  that 
influence  the  regulation  of  metacognitive  behaviors.  The  ultimate  goal  is  to  support 
and  influence  a  learner’s  approach  to  problem  solving  and  learning  in  general. 
While  the  research  identifies  multiple  examples  of  successful  strategy 
implementation,  what  the  literature  lacks  are  guidelines  for  when  best  to  instantiate 
them  based  on  the  domain  being  trained  and  the  environment  the  interaction  is 
taking  place  within.  To  enhance  GIFT’S  authoring  functionalities,  a  generalized 
ontology  is  required  that  links  specific  instructional  strategies  and  techniques  with 
high-level  domain-relevant  content  along  with  the  types  of  tutoring  environments 
and  the  services  they  can  afford.  By  defining  these  relative  dependencies,  an  ITS 
developer  can  embed  empirically  recommended  metacognitive  tutoring  functions 
based  on  characteristics  associated  with  the  content  being  produced. 

In  the  context  of  tutor  development  within  the  GIFT  architecture,  the  goal  is  to 
establish  a  set  of  strategies  the  framework  can  support  and  conceptualize  their 
application  for  determining  future  authoring  requirements  to  support  their 
implementation,  both  from  a  modeling  and  pedagogical  delivery  standpoint.  In 
accordance  with  this  subvector  of  instructional  management  research  in  adaptive 
training  environments,  research  questions  of  interest  include  the  following: 

•  What  modeling  implications  are  associated  with  supporting  instructional 
management  to  develop  metacognitive  skills? 

•  What  architectural  modifications  are  required  to  infer  metacognitive 
relevant  states  from  raw  interaction  data  outputs? 

•  What  combination  of  assessment  techniques  provides  the  most  accurate 
prediction  of  metacognitive  strategy  application? 

•  What  metacognitive  strategies  can  be  inherently  assessed  across  all  domains 
regardless  of  the  training  application? 

•  How  do  metacognitive  strategies  translate  across  domains  and  training 
environments? 

•  What  educational  data  mining  techniques  can  be  applied  to  infer  self- 
regulated  learning  abilities  from  common  GIFT  produced  log  files? 

•  How  do  scaffolding  practices  of  delivery  and  fading  for  domain-specific 
guidance  functions  translate  to  metacognitive  tutoring? 
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Can  metacognitive  tutoring  models  and  strategy  implementations  be 
developed  by  nontechnical  SMEs? 


6.4  Optimizing  the  Selection  of  Tactics 

Our  fourth  goal  is  to  optimize  the  selection  of  tactics — domain-specific  actions  by 
the  tutor — to  provide  the  greatest  opportunity  for  performance,  learning,  retention, 
and  transfer.  In  GIFT,  tactics  are  the  actions  taken  by  the  tutor  in  response  to  learner 
states  and  instructional  context  (e.g.,  conditions  of  the  scenario  or  problem 
presented),  as  shown  in  Fig.  2,  and  are  constrained  by  available  options  provided 
during  the  authoring  process.  Improving  the  usability  and  efficiency  of  authoring 
tools  will  likely  result  in  a  greater  number  of  available  options  for  adaptive  training 
domains. 


Unlike  instructional  strategies,  which  are  derived  from  good  pedagogical  practices- 
based  learning  theory  and  influenced  by  the  learner’s  states,  tactics  are  domain- 
specific  actions  by  the  tutor  and  may  not  be  generalized  across  all  task  domains. 
Research  is  needed  to  determine  methods  to  select  the  best  possible  tactic  given  the 
selected  instructional  strategy,  the  training  domain,  and  the  availability  of  tactics. 


Modeling  the  expert  behaviors  of  human  tutors  may  be  a  starting  point  but  accurate 
assessment  methods  are  needed  for  both  individual  and  team  level  states.  These 
states  are  critical  in  selecting  appropriate  strategies  (plans  for  action)  and  tactics 
(actions;  e.g.,  assessments,  feedback,  questions,  and  changes  to  the  training 
environment)  per  the  Learning  Effect  Model  (Sottilare  et  al.  2013b)  as  shown  for 
both  individuals  (Fig.  2)  and  teams  (Fig.  4).  Assessment  of  team  states  may  also  be 
useful  in  determining  constraints  to  be  monitored  by  tutoring  agents  and 
interactions  with  the  learner  and  the  training  environment  as  shown  in  Fig.  2. 
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6.5  Personalization  (Occupational  and  Noncognitive  Factors) 

Personalization  of  content  to  an  individual  student’s  interests  has  been  shown  to 
have  a  positive  impact  on  learning,  retention  of  infonnation,  and  motivation 
(Cordova  and  Lepper  1996).  Further,  using  this  strategy  has  also  demonstrated 
positive  effects  in  transfer  performance,  which  is  the  ability  to  extend  what  was 
learned  to  related  tasks  (Walkington  2013).  This  instructional  strategy  is  generally 
referred  to  as  context  personalization,  and  has  been  shown  to  be  effective  in 
adaptive  computer  environments  (Anand  and  Ross  1987;  Cordova  and  Lepper 
1996;  Walkington  2013). 

The  majority  of  the  context  personalization  research  has  been  in  the  area  of 
mathematics,  with  learning  and  transfer  gains  being  found  in  areas  such  as  algebra, 
fractions,  and  word  problems  (Anand  and  Ross  1987;  Cordova  and  Lepper  1996; 
Ross  et  al.  1986;  Walkington  2013).  The  self-reference  effect,  or  the  ability  for 
reference  to  the  self  to  improve  memory  (Symons  and  Johnson  1997),  is  often 
studied  along  with  context  personalization  by  including  the  student’s  name  or 
personal  information  in  the  materials  (Anand  and  Ross  1987;  Cordova  and  Lepper 
1996;  Sinatra  et  al.  2014).  In  addition  to  mathematics,  the  self-reference  effect  has 
been  examined  in  domain  areas  such  as  science  and  deductive  reasoning  (Moreno 
and  Mayer  2000;  Sinatra  et  al.  2014). 

By  personalizing  the  context  of  the  material  that  is  learned,  it  could  have  both 
affective  and  cognitive  impacts  on  the  individual.  Motivation  and  positive  attitudes 
toward  the  material  has  been  shown  to  increase  in  personalized  contexts  (Anand 
and  Ross  1987;  Cordova  and  Lepper  1996).  In  an  adaptive  system  with  self-entered 
preferences,  the  student  may  feel  that  the  system  is  taking  his  or  her  preferences 
into  account  if  interest  personalized  questions  are  given,  which  may  result  in  more 
positive  feelings  toward  the  system  (Ritter  et  al.  2007).  Additionally,  by  aligning 
examples  and  information  in  the  content  to  be  learned  with  the  previous  knowledge 
base  of  the  individual,  it  may  both  increase  interest  in  the  training  and  lessen  the 
mental  workload  required  to  learn  the  material  (Anand  and  Ross  1987;  Ku  and 
Sullivan  2002;  Ross  1983;  Sinatra  et  al.  2014). 

Research  has  suggested  that  by  using  examples  that  are  consistent  with  one’s 
college  major  and  area  of  expertise  it  can  improve  learning  outcomes.  Specifically, 
it  was  found  that  nursing  students  who  received  medical  examples  had  more 
positive  outcomes  than  those  who  were  given  teaching  examples.  Conversely, 
education  students  who  were  given  teaching  examples  on  the  same  task  performed 
better  than  when  they  were  given  medical  examples  (Ross  1983).  Follow-up  work 
found  that  both  individually  selected  interest  preferences  and  general  adaptation  to 
interests  (based  on  major)  showed  positive  learning  outcomes  (Ross  et  al.  1986). 
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The  reason  behind  these  effects  may  be  2-fold:  By  using  examples  that  are 
consistent  with  one’s  knowledge  base,  it  may  be  inherently  more  interesting  to  the 
individual;  further,  it  may  reduce  the  amount  of  cognitive  effort  that  is  necessary  to 
engage  with  the  material  since  the  student  already  has  a  good  understanding  of  the 
context. 

It  would  be  advantageous  to  continue  to  research  into  the  impact  of  context 
personalization  in  adaptive  training  programs.  Current  ITS  systems,  such  as  GIFT, 
offer  more  flexibility  and  features  than  systems  and  computer  programs  that  were 
developed  in  the  original  context-personalization  research  conducted  in  the  1980s 
and  1990s.  Additionally,  as  a  domain-independent  framework,  GIFT  can  be  used 
to  examine  the  impact  of  context  personalization  in  a  variety  of  other  domains, 
whereas  in  the  past  the  research  has  primarily  focused  on  math  instruction.  One 
approach  to  context-personalization  research  that  can  be  taken  with  GIFT  is  to  do 
work  similar  to  Ross  (1983),  in  which  the  context  of  the  problems  and  materials  are 
specifically  matched  or  mismatched  with  the  individual  learner’s  specialty  area.  In 
the  context  of  military  training,  a  Soldier’s  MOS  and  near-term  assignments  can  be 
used  to  personalize  a  training  experience  to  better  prepare  that  individual  for  the 
environment  they  will  be  operating  within. 

Rather  than  using  mathematics  as  the  domain  of  interest,  a  military-relevant  domain 
can  be  chosen.  Providing  materials  that  are  matched  to  the  individual  learner’s 
specialty  area  is  expected  to  have  a  positive  impact  on  learning  and  attitudes  toward 
the  experience.  Learning  outcomes  are  expected  to  be  improved,  as  the  individual 
will  not  already  have  an  understanding  of  the  context  of  the  provided  examples  but 
also  will  be  able  to  easily  see  why  it  is  relevant  to  their  own  job. 

To  support  personalization,  additional  studies  could  examine  the  impact  of 
allowing  learners  to  select  the  context  of  the  questions  they  will  receive  based  on 
their  own  preferences  or  the  task  that  they  will  be  engaging  in.  In  many  military- 
related  tasks  there  are  subtle  differences  in  the  task  that  will  be  performed  based  on 
the  geographic  location  of  their  assignment.  For  instance,  if  an  individual  is  tasked 
with  interacting  and  negotiating  with  individuals  from  a  culture  other  than  their 
own,  they  may  engage  with  a  negotiation  tutor.  However,  depending  on  the  culture 
that  they  are  to  engage,  there  may  be  different  phrases  or  customs  that  should  or 
should  not  be  used.  The  basic  elements  of  negotiation  will  be  similar  but  the 
questions  and  materials  can  be  edited  to  have  geographic  and  culturally  specific 
examples  that  will  be  more  consistent  with  the  actual  experience  the  individual  will 
have.  Research  can  be  conducted  on  the  level  and  types  of  material  and  assessment 
personalization  that  results  in  positive  outcomes  and  performance. 
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To  support  this  type  of  research  and  personalization,  ideally  in  an  ITS,  general 
questions  can  be  created,  and  based  on  learner  entered  or  selected  infonnation  they 
can  be  personalized  for  the  individual  student.  The  following  is  a  math  example  to 
demonstrate  simple  question  context  personalization: 

•  General  question:  “Y ou  have  4 _ .  Y ou  are  offered  5 _ in  exchange 

for  half  of  your _ .  How  many _ do  you  have  after  the  trade?” 

•  If  you  were  using  the  context  of  fruit,  the  problem  could  read  “You  have  4 
apples.  You  are  offered  5  oranges  in  exchange  for  half  of  your  apples.  How 
many  pieces  of  fruit  do  you  have  after  the  trade?” 

•  Additionally,  if  you  were  using  the  context  of  books,  the  question  could 
read  “You  have  4  science  fiction  novels.  You  are  offered  5  mystery  novels 
in  exchange  for  half  of  your  science  fiction  novels.  How  many  novels  do 
you  have  after  the  trade?” 

In  this  example,  a  general  question  is  authored  and  then  specific  items  are  added  in 
to  change  the  context  or  topic  of  the  question.  Research  can  examine  the  impact  of 
providing  questions  and  examples  that  are  consistent  (and  inconsistent)  with  the 
learner’s  job  or  interest  areas.  To  do  so  in  GIFT,  edits  may  be  needed  to  be  made 
such  that  GIFT  could  store  a  bank  of  questions  that  are  similar  to  each  other  but 
differ  in  context.  The  version  of  the  questions  that  are  consistent  with  the  learner- 
selected  context  can  then  be  selected  by  the  system  and  used  during  training. 
Additional  techniques  for  personalization  of  questions  and  materials  may  include 
allowing  for  user-entered  infonnation  (such  as  the  individual’s  name)  to  be 
integrated  into  the  questions. 

If  these  or  similar  studies  are  shown  to  be  successful,  multiple  versions  of  training 
question  domains  can  be  generated  in  ITSs,  resulting  in  the  learner  engaging  with 
the  one  that  is  most  relevant  to  themselves  based  on  previous  experience  and  self- 
entered  preferences.  As  functionality  is  being  developed  in  ITSs  to  track  student 
learning  history  and  characteristics,  it  may  also  be  advantageous  to  store  student 
preferences,  interests,  and  the  expertise  area  of  the  individual.  This  stored 
information  can  then  be  used  for  context  personalization  to  ultimately  increase 
learner  performance  and  retention  in  ITSs  such  as  GIFT. 

While  personalization  is  an  instructional  strategy,  research  based  on  it  will 
ultimately  impact  both  the  areas  of  authoring  and  learner  modeling  in  GIFT. 
Authoring  tools  and/or  support  will  need  to  be  added  into  GIFT  to  allow  for 
personalized  questions  and  storing  personalized  question  banks.  Storage  of  learner 
preferences  and  information  will  need  to  be  stored  by  GIFT’S  learner  module  such 
that  they  can  be  used  in  future  trainings  and  engagements  with  the  system. 
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The  following  are  personalization-related  research  questions: 

•  Will  matching  the  context  of  materials  to  be  consistent  with  a  learner’s  job 
or  interest  area  lead  to  increased  learning  and  performance? 

•  Will  individuals  who  receive  context  personalized  materials  have  more¬ 
positive  attitudes  toward  the  tutoring  system? 

•  What  types  of  personalization  (e.g.,  based  on  self-selected  interest,  based 
on  job,  and  based  on  task)  will  have  the  best  outcomes? 

•  Are  their  individual  differences  that  will  moderate  the  benefits  of 
personalization?  If  so,  what  types  of  learner  characteristics  should  result  in 
the  use  of  personalized  materials? 


7.  Interdependencies  with  Other  Adaptive  Training  Research 
Vectors 


This  section  examines  interdependencies  between  instructional  management  and 
the  other  5  adaptive  training  research  vectors  (Fig.  5).  This  discussion  forms  the 
basis  for  the  sequencing  of  research  and  ultimately  bringing  adaptive  training 
capabilities  into  a  state  of  practice. 
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Accurate  methods  to  classify  individual  and  team  learner  states  are  a  necessary 
precursor  to  selecting  optimal  instructional  strategies,  as  noted  in  the  learning  effect 
models  for  individual  learners  (Fig.  2)  and  teams  of  learners  (Fig.  4).  In  turn, 
instructional  strategies  along  with  instructional  context  are  necessary  precursors  to 
selecting  optimal  instructional  tactics  and  ultimately  significant  effect  on  desired 
outcomes:  learning,  performance,  retention,  and  transfer. 

As  such,  it  is  difficult  to  conceptualize  the  application  of  instructional  management 
focused  research  without  involving  other  vector  components  of  the  GIFT  program. 
Therefore,  the  dependency  of  instructional  management  research  on  other 
functional  areas  of  the  architecture  is  significant. 

7.1  Learner  Modeling  and  Instructional  Management 

Adaptive  training  systems  are  learner-centric  systems.  Independent  of  the  domain 
under  training,  accurate  modeling  of  the  learner  is  critical  to  driving  instructional 
decisions  in  adaptive  training  systems.  However,  collection  and  maintenance  of  this 
data  may  be  costly,  so  it  is  necessary  to  select  measures  and  states  that  significantly 
impact  our  desired  outcomes:  learning,  performance,  retention,  and  transfer. 
Research  is  needed  to  determine  what  this  dataset  will  look  like.  Candidates  abound 
in  the  literature  but  in  general  these  include  transient  data/states,  cumulative  states 
(building  over  time),  and  enduring  data/states  (Paneva  2006).  Learning  falls  into 
both  transient  and  cumulative  states.  It  is  necessary  to  understand  progress  toward 
domain  competency  in  addition  to  measures  of  near-tenn  perfonnance,  and  it  is 
important  to  understand  how  domain-specific  learning  (skills)  decay  over  time. 

Transient  measures  of  importance  include  individual  behavioral  and  physiological 
data,  and  cognitive,  affective,  physical,  and  social  states  to  represent  learning. 
Cumulative  measures  include  achievements  (e.g.,  certifications,  training, 
education,  and  experiences),  affiliations,  work  history,  and  domain  competency. 
More-enduring  information  about  the  learner  might  include  gender,  culture,  first 
language,  physical  constraints  (e.g.,  colorblind/deaf),  values,  personality  attributes, 
or  other  trait-based  infonnation  (Sottilare  and  Brawner  2014).  All  of  these 
measures/states  are  potential  drivers  for  adaptive  training  decisions. 

In  terms  of  instructional  management,  sound  pedagogy  is  reliant  on  accurate 
representation  in  the  learner  model.  It  is  important  to  identify  variables  in  a  learner 
model  that  moderate  learning,  account  for  variance  in  perfonnance,  and  can  be 
accurately  assessed  in  real  time  for  informing  strategy  selections.  Selecting  an 
inappropriate  strategy  and  tactic  combination  can  result  in  negative  learning 
outcomes  and  can  ultimately  impact  the  perceived  value  of  the  system. 
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7.2  Domain  Modeling  and  Instructional  Management 

In  GIFT,  instructional  management  takes  place  in  2  modules/processes  within  the 
learning  effect  model.  One  process  is  instructional  strategy  selection  within  the 
pedagogical  module.  The  second  is  within  the  domain  module  where  specific 
tactics  or  actions  are  selected  based  on  the  strategy  selection  and  instructional 
context.  An  important  component  of  instructional  management  is  translating  a 
generalized  strategy  into  a  tactic  that  can  be  executed  within  a  specific  training 
environment.  This  requires  understanding  what  knowledge  components  make  up  a 
domain  and  what  tools  are  available  to  guide  a  learner  and  adapt  the  training  event. 

In  addition,  domain  modeling  plays  a  critical  role  in  enabling  the  use  of  reusable 
learning  objects.  When  applying  instructional  management  practices  in  an  outer- 
loop  capacity  through  GIFT’S  EMAP  (Fig.  3),  a  well-designed  domain  model  can 
be  used  to  identify  content  that  can  be  presented  to  a  learner  along  with  data  that 
supports  its  application.  This  supports  ease  of  authoring  as  well,  as  a  developer  can 
leverage  existing  content  if  the  domain  model  has  overlap  with  existing  course 
representations. 

7.3  Authoring  Tools  and  Instructional  Management 

Authoring  tools  and  methods  are  needed  to  configure  instructional  management 
practices  across  all  adaptive  training  systems  and  domains  of  instruction.  There  are 
2  types  of  authors  that  must  be  considered.  First,  training  developers  and  SMEs  are 
targeted  as  end-users.  These  individuals  will  use  GIFT  services  to  create  highly 
personalized  adaptive  training  events  by  leveraging  the  pedagogical  models 
established  within  available  baseline  versions.  While  a  training  developer  and  SME 
are  well  versed  in  the  knowledge  components  of  a  domain,  they  will  lack  many  of 
the  skills  required  to  build  adaptive  training  solutions.  The  authoring  tools  must  be 
designed  to  compensate  for  a  developer’s  lack  of  understanding  across  the  various 
disciplines  involved  in  creating  such  systems.  Building  tools  that  guide  the  various 
authoring  processes  is  critical  to  the  success  of  GIFT  outside  of  the  laboratory 
setting.  It  will  also  be  important  that  these  tools  help  individuals  maintain  a 
contextualized  awareness  of  what  they  authoring  and  how  that  will  be  translated 
into  a  run-time  instantiation. 

The  second  author  to  consider  is  a  research  scientist  interested  in  instructional 
management  research  within  adaptive  training  systems.  These  individuals  will 
require  the  ability  to  manipulate  various  components  of  the  GIFT  modules  to 
support  their  research.  Using  the  ATLEC  as  a  guiding  function  to  highlight 
instructional  management  research  in  ITSs,  authoring  processes  must  be 
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established  to  configure  the  inputs,  processes,  and  outputs  of  the  pedagogical 
module.  This  includes  defining  the  variables  from  the  learner  model  that  moderate 
strategy  selection  (with  a  large  assumption  that  the  modeling  techniques  applied  for 
that  variable  are  accurate),  defining  the  logic  and  policies  that  dictate  what  strategy 
to  select  based  on  learner  data,  and  defining  how  that  strategy  is  translated  into  a 
specific  tactic  to  be  executed  in  the  environment.  This  is  greatly  different  from  the 
training  developer/SME  example,  as  the  only  intended  authoring  function  of  this 
group  is  defining  the  tactic  that  would  be  executed. 

Another  avenue  of  authoring  that  must  also  be  addressed  is  metadata,  which  plays 
a  large  role  in  instructional  management  within  a  domain-independent  framework, 
as  it  provides  a  way  to  describe  your  content  and  guidance  functions  in  a  relatively 
generic  fashion.  In  terms  of  authoring,  tools  must  be  developed  to  facilitate  easy 
application  of  metadata  to  learning  objects.  Research  is  required  to  determine  how 
best  to  support  this  function. 

7.4  Evaluation  and  Instructional  Management 

Training  effectiveness  research  is  focused  primarily  on  the  impact  of  instructional 
management  practices  on  learning  outcomes  of  interest:  perfonnance,  retention, 
and  transfer.  To  optimize  the  application  of  adaptive  training  in  the  Anny, 
automated  training  effectiveness  tools  are  required  to  run  analyses  as  data  become 
available.  Much  of  these  capabilities  are  researched  within  communities  dedicated 
to  educational  data  mining.  The  goal  is  to  test  pre-established  configurations  to 
detennine  their  impact  on  learning  and  adapt  instructional  management  policies  to 
better  optimize  learning  experiences. 

A  goal  of  instructional  management  in  GIFT  is  to  optimize  perfonnance  and 
competency  outcomes  through  personalized  training  experiences  that  adapt  to  an 
individual’s  KSAs.  A  connection  between  big  data  and  instructional  management 
is  applying  large  datasets  from  prior  course  interactions  to  update  and  improve 
technique  and  strategy  implementations  across  ah  available  courses.  A  challenge 
with  defining  instructional  management  logic  in  a  domain-independent  context  is 
that  it  requires  generalizability  across  applications.  An  issue  with  instructional 
strategy-based  research  is  that  an  approach  taken  in  one  domain  is  hard  to  translate 
to  a  different  one  without  performing  extensive  research  to  validate  its  application. 
In  addition,  it  is  difficult  to  define  definitive  instructional  management  logic  based 
on  these  uncertainties.  As  such,  big  data  can  be  used  to  account  for  this  uncertainty 
by  applying  machine  learning  and  data  mining  techniques  to  assess  specific  causal 
relationships  between  instructional  practices  and  outcomes  on  perfonnance, 
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retention,  and  transfer.  This  application  of  big  data  is  used  to  reinforce  instructional 
management  models  by  optimizing  itself  over  time  as  more  and  more  data  are  made 
available. 

7.5  Architecture  and  Instructional  Management 

Managing  instructional  strategy  selection  and  tactic  delivery  depends  upon  multiple 
components  of  GIFT.  This  associates  domain  modeling  to  apply  context  to  a 
pedagogical  decision,  learner  modeling  to  provide  trainee  relevant  information  that 
triggers  a  pedagogical  intervention,  authoring  to  provide  a  means  for  building  these 
linkages  and  representations,  and  training  effectiveness  to  determine  if  a  strategy 
or  set  of  strategies  had  an  effect  on  performance-related  outcomes.  This  highlights 
an  important  point;  while  each  of  the  aforementioned  components  of  instructional 
management  has  separate  processes,  the  architecture  is  the  component  that  dictates 
implementation  design  and  development. 

In  terms  of  architecture,  GIFT  end-state  goals  require  potential  integration  with  a 
number  of  technologies  that  facilitate  varying  roles  of  instructional  management 
practices.  These  technologies  include  tools  and  methods  to  support  content 
management,  natural  language  processing,  text-to-speech  processing,  virtual 
human  authoring  and  configuration,  social  media  framework  connections,  and 
training  application  manipulations  (e.g.,  manipulating  the  weather  in  a  virtual 
world).  In  addition,  specific  architectural  modifications  will  be  required  to  perform 
tasks  inherent  to  the  current  standards  of  GIFT,  including  methods  to  create 
messaging  templates  used  to  auto-populate  feedback  scripts  with  context  relevant 
information  established  in  log  files,  the  ability  to  personalize  strategy  selections  on 
an  outer-  and  inner-loop  capacity  across  learners  and  teams  of  learners,  and  the 
application  of  actionable  metadata  and  API  statements  to  appropriately  link  learner 
information  and  prior  experience  with  appropriate  training  and  optimized 
configurations.  In  dealing  with  a  domain-agnostic  intelligent  framework  such  as 
GIFT,  the  use  of  machine  learning  and  data  mining  techniques  is  required  to 
reinforce  and  optimize  pedagogical  logic  over  time. 

8.  Conclusions 


This  report  outlines  the  ARL’s  plans  for  conducting  research  in  adaptive  training 
and  education  to  support  the  ALM.  Specifically,  this  report  relates  to  instructional 
management  and  the  answer  to  the  following  question:  What  adaptive  instructional 
methods  (strategies,  tactics,  and  techniques)  are  most  effective  for  individual  and 
team-based  military  training  and  educational  domains? 
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This  report  outlined  the  following  goals: 


•  Model  the  perception,  judgment,  and  behaviors  of  expert  human  tutors  to 
support  practical,  effective,  and  affordable  learning  experiences  guided  by 
computer-based  agents. 

•  Leverage  prior  research  in  the  learning  sciences  and  AI  communities  (both 
theoretical  and  empirical)  to  establish  a  set  of  best  practices  on  how  to 
author  and  execute  techniques,  strategies,  and  tactics  across  any  domain  of 
instruction. 

•  Discover  and  develop  modeling  functions  that  account  for  uncertainty 
across  policies  informing  pedagogical  decisions  (e.g.,  content  delivery, 
course  navigation,  and  guidance). 

•  Personalize  instructional  technique  and  strategy  selections  based  on 
individual  differences  infonned  through  empirical  evaluations  and 
reinforcement  machine  learning  methods. 

•  Develop  authoring  tools  grounded  in  cognitive  and  instructional  theory  and 
informed  by  empirical  research  to  assist  training  managers,  developers,  and 
SMEs  in  building  pedagogically  sound  training  experiences  without  the 
requirement  of  programming. 

•  Support  individual  and  team  training  (e.g.,  small  unit  and  collective 
training)  and  education  (e.g.,  collaborative  learning  and  problem-solving) 
experiences. 
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List  of  Symbols,  Abbreviations,  and  Acronyms 


c 

sigma 

AAR 

after-action  review 

AI 

artificial  intelligence 

ALM 

US  Army  Learning  Model 

ARL 

US  Army  Research  Laboratory 

ATLEC 

Assistive  Technology  Learning  through  a  Unified  Curriculum 

DIS 

distributed  interactive  simulation 

EMAP 

Engine  for  Management  of  Adaptive  Pedagogy 

GIFT 

Generalized  Intelligent  Framework  for  Tutoring 

ITS 

intelligent  tutoring  system 

KSA 

knowledge,  skill,  and  ability 

LEM 

Learning  Effect  Model 

MOS 

military  occupational  specialty 

QandA 

question  and  answer 

SME 

subject  matter  expert 

SRL 

self-regulated  learning 

TandE 

training  and  education 

BVS3 

Virtual  Battlespace  3 

WFO 

Warfighter  outcome 

ZPD 

zone  of  proximal  development 
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